Method for automatically handling, assembling and working on objects

ABSTRACT

Method and apparatus are disclosed for determining the position and attitude of objects, together with robotic systems employing same, and methods of targeting objects for such purposes. 
     Particularly of interest is the application to continuous conveyors, particularly in the process of automobile manufacture.

FIELD OF THE INVENTION

This application is a continuation of application Ser. No. 348,803 filedFeb. 16, 1982 now abandoned.

This invention relates to methods and apparatus for determining theposition of an object and guiding robots or other automation to handleor work on said object. It is a continuation in part of my previousapplication Ser. No. 200,401 entitled "Fiber Optic Based RobotControls".

There are many instances in which it is desired to know the position ofan object. In the manufacturing field, such instances include theposition of objects along mass production lines, particularly thosewhich are highly automated. For example, in a mass production line, itis frequently necessary to know with considerable precision the positionof an object suspended from a conveyor system. This is particularly truein automated systems involving the use of robots for it is fundamentallynecessary that the object be in a known position, relative to the robot,before the robot can execute a desired manipulation of the object.

In some instances, mechanical means are used to position the objectand/or to orient the object in proper position for automatedmanipulation. For many applications, however, it is necessary to provide"vision" in order to determine the position of an object. This isparticularly true in the case of robots which can perform a myriad ofphysical manipulations, all automatically. It is well recognized in thefield of robots that robotic "vision" is one of the major obstacles tomuch wider use of robots which are presently quite sophisticated interms of the manipulations of which they are capable.

Specifically the problem in plants of operationally using robots tohandle or work on random parts on continuous conveyors is an enormousone. Since such conveyors are omni present in plants of all types, thisproblem must be effectively dealt with, if large scale robot usage is tobecome a reality.

In doing so, there are many types of electro-optically based "machinevision" systems which can be utilized. Historically, these systems havebeen based on reflective viewing of objects through their gray scalelevels which poses extremely difficult problems. The trend is thence toever more complex systems, which runs counter to good plant reliability.

This inventor, for example, has been involved in the installation ofnearly 1,000 electro-optical sensor units in plants of varying types forinspection. Substantial difficulties have been encountered when suchelectro-optical image based sensors were utilized to obtain part images,particularly in reflection.

When one considers that the robot based system must achieve areliability far higher than even these inspection based units, in orderthat it not ruin the product, drop it on the floor etc., it becomesapparant that a simple and reliable means of solving these problems isrequired.

This invention seeks to illustrate such means. In particular, solutionis possible if one restricts the problem simply enough to targetedobjects. This then leads to the possibility of tracking the parts or thecontainers, conveyors etc. so targeted, possibly using further sensorsif required to find parts within these containers, instrumented grippersor the like.

BACKGROUND

The recent application by the inventor, Ser. No. 200,401, of which thisapplication is a continuation-in-part, illustrated in the embodiment ofFIG. 13, instrumented monorail and walking beam conveyors utilizingfiber optics directed through portions of the conveyor apparatus whichcould be illuminated on demand in order to provide one or more targetsfor tracking or homing purposes using robotic or other automation.

Also described in Ser. No. 200,401 are many other novel features ofinterest. These are:

(a) The general concept of use of such `active` lighting in automationand particularly the use of fibers therefore.

(b) The use of `active detection` wherein the light is directed from therobot into one end of a fiber, and sensed at the opposite end of saidfiber.

(c) The use of other materials than fibers, for example, translucentfixtures of teflon or ceramic.

(d) The use of multiple target points on the illuminated piece to betracked.

(e) The use of blow-offs to keep the targeted fixtures clean.

(f) The use of pulsed or modulated light sources discrimination againstbackground noise.

(g) The use of light sources and electro-optical sensors both located onrobot where the light source of the robot is directed to a predictedentrance point of the fiber(s) and the light emanating from the oppositeend of the fiber(s) is sensed by the camera of the robot. Two robots forexample could be used, one to light the part or fixture, the other tosense it.

In addition, the copending application described how to track conveyorscarrying parts and made reference to the tracking the parts themselves.

OBJECT OF THE INVENTION

It is an object of the present invention to provide a method andapparatus for determining the position of a targeted object or objectcarrier. It is a further object of the invention to provide such methodand apparatus having particular suitability for providing robotic"vision" and to disclose practical system based thereon.

It is further intent of this invention to expand the previouslydisclosed concepts beyond simply the fixtures themselves and the partswithin the fixture, to the targeting and identifying of containers andparts of all descriptions, for example to baskets, trays, cartons, toolsfor pickup, parts in warehouse bins, and indeed the parts themselves.

In addition, this invention elaborates further on the use of othertargeting materials than fiber optics, not only transmissive materialsof other sorts, but objects such as glass beads, drilled fascets,casting risers and the like.

It is a further purpose of this invention to show other means in thesensing which can give improved target position and data. Particularlyof interest are those related to apparatus such as Pinkney, U.S. Pat.No. 4,219,847.

It is further a purpose of this invention to show specifically howcertain parts in overhead monorail conveyors can be picked off underrandom situations with the high reliability needed to work in plants.

It is further a purpose of this invention to show means for outliningthe edges of targets for use with stereo cameras or other sensing meansnot necessarily based on point targets.

It is a further purpose of this invention to show means for coding thevarious fiber input or outputs, or other targets by use of colors ormodulation frequencies. This is also possible with inserted glass beadsand retro-reflectors.

Finally, it is a desirable purpose of this invention to show thatstandardized systems, based on such tracking, can be used across thetotal spectrum of manufacturing and industry with very litte change aslong as one sticks to certain target principles. This allows the widespread use of reliable guide robots at affordable cost.

These and other purposes of the invention will become clear onconsideration of the following embodiments:

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

There follows a detailed description of preferred embodiments includingthe drawings in which:

FIG. 1 is a diagrammatic side elevation view of an overhead conveyorsystem utilizing the present invention,

FIG. 2 is an enlarged diagrammatic perspective view of hook 14 of FIG.1.

FIG. 3A, 3B illustrates a complete system for the picking oftransmission clutch parts off of overhead monorail conveyors in anautomatic transmission plant. The sensor is of the general typedescribed by Pinknewy, utilized to track the conveyor carrier(s) whichare targeted by means herein disclosed. Optionally, an additional sensoror analysis means on the robot may be used to find the part within thecarrier.

FIG. 4 illustrates target embodiments on the carriers used in FIG. 3.

FIG. 5A, 5B illustrates an embodiment showing methods of targeting aplastic door panel having built-in optical fibers.

FIG. 6 illustrates a car windshield with lossy fibers to outline itsperiphery, which is imaged by stereoscopic cameras.

FIG. 7A,7B illustrates a part targeting embodiment employing drilledholes.

FIG. 8A, B, C illustrates a part targeting embodiment wherein saidtargets are cast into the part or are appendiges thereto.

FIG. 9 illustrates a part targeting embodiment wherein said targetscomprise directional or color reflective elements which may be molded orpressed in.

FIG. 10 illustrates a robotic system employing targeted boxes randomlyspaced on a roller conveyor, utilizing either targets printed onto thebox, fibers, or retro-reflectors.

FIG. 11 illustrates means for targeting a tool.

FIG. 12 illustrates alternative means for creating siitable targets, anduse on car body assembly.

FIG. 13 illustrates reusable targets for parts.

FIG. 14 illustrates a method of assembling cars according to theinvention.

FIG. 15 illustrates a sensor embodiment according to the invention.

FIGS. 16A, B, C and 17 illustrate further part targeting embodiments.

Incorporated by reference other copending applications by the inventor.Many embodiments therein illustrate sensors and hardware of use in thisdisclosure.

1. Electro-optical sensor systems for thread and hole inspection Ser.No. 64,867

2. Method and apparatus electro-optically determining the dimension,attitude and location of objects: Ser. No. 34,278

3. Method and apparatus for determining physical characteristics ofobject and object surfaces: Ser. No. 15,792

4. New photodetector array based optical measurement systems: Ser. No.163,290

5. Electro-optical inspection, Ser. No. 073,226

6. Co-ordinate measuring method and device, Ser. No. 201,081

7. Electro-optical sensors with fiber optic bundles, Ser. No. 173,370

8. Electro-optical surface roughness measurement and control Ser. No.240,459

9. Apparatus for determining dimensions, Ser. No. 134,465

10. High speed electro-optical inspection, Ser. No. 203,866

11. Fiber optic based robot controls, Ser. No. 200,401

12. Electro-optical sensors for machine tool and robotic inspection.

13. Electro-optical systems for control of robots, manipulator arms andcoordinate measurement machines Ser. No. 262,497

14. Method and apparatus for determining wear or breakage of tools andother defects

15. Electro-optical systems for detection of leakage and blockage

16. Productivity improvement via robotic electro-optical part and toolinspection

17. Robot tractors

18. Method and apparatus for determining physical characteristics ofobject outer surfaces Ser. No. 15,614

19. Method and apparatus for determining dimensional informationconcerning an object (division of Ser. No. 15,792)

20. Method and apparatus for detection of surface deformaties (divisionof Ser. No. 15,792) Ser. No. 234,728

21. "Linear" continuation of Ser. No. 015,792

22. "Circular" continuation of Ser. No. 015,792

23. Optically controlled plumbing apparatus Ser. No. 29,840

24. Optically controlled bathing systems Ser. No. 23,150

25. Electro-optical and robotic casting quality assurance

26. Controlled machining of combustion chambers, gears and othersurfaces including methods for obtaining correct combustion chambervolume in finished engine assemblies.

With reference to FIG. 1, an item 10 is shown being moved in thedirection of arrow A below an overhead conveyor system 11 generallycomprising a track 12 on which a carriage 13 is movable. Object 10 issuspended in any suitable manner from a hook 14 secured to carriage 13.Hook 14 includes an aperture 15 for a bolt 16 for securing hook 14 tocarriage 13. A fiber optic 17 is embedded within hook 17 and has a lightreceiving end 18 in a surface 19 of hook 14 and a light emitting end 20in a surface 21 of hook 14. Fiber optic 17 can be any fiber opticelement such as a single fiber optic or a bundle thereof, of whichseveral are commercially available. A plastic `corfon` fiber opticelement is quite suitable.

In FIG. 1, it is assumed that it is desired to determine the position ofitem 10 suspended from hook 11 when item 10 is at a general locationdesignated 22, in FIG. 1. It is also assumed in FIG. 1 that item 10 isin a known position relative to the position of hook 14.

A light source 23 is positioned above track 12 to direct lightdownwardly such that it will be incident on the upper surface 19 of hook14 of carriage 13 when a carriage is positioned below the light source.

A light detector 24, in this case a scanning matrix photo detectorcamera comprising a lens and a detector array comprised by a pluralityof horizontal rows of discrete photo diodes, is positioned adjacent theconveyor so as to be adjacent hook 14 when conveyed item 10 is locatedgenerally in position 22. More particularly, when the conveyed item isin position 22, the light emitting end 20 of fiber optic 17 is imaged bythe camera lens 40 onto the matrix array which provides real timeinformation as to the location of hook 24.

In a typical case, illustrated in FIG. 1, the light 25 emitted from end20 is imaged to form a spot 25 on four adjacent photo diodes of array24. As the photo diode array is scanned, an output signal 26, indicativeof the position of the spot of light 25 on array 24 is conveyed tosuitable means 27, such as microcomputer, to determine the position ofhook 14, and thus item 10, relative to any known position, such as theposition of a robot, or the position of detector array 24. A signal 28,indicative of the position of hook 14 and/or item 10 is then conveyed tosuitable robot control means for control of a robot, not shown, formanipulation of the hook 14 of conveyed item 10. A signal 29 controllingthe robot thus includes positional information concerning the hook 10 oritem 14 to be manipulated by the robot.

It will be readily apparent that the matrix array in FIG. 1 providespositional information in the x and y directions in the plane of thedrawing. It is also possible to readily provide information concerningposition in the z axis. For example, as shown in dotted lines in FIGS. 1and 2, a further fiber optic element 30 may be embedded in hook 14,extending from horizontal upper surface 19 to front surface 31 whichextends vertically and transverse to the plane of the drawing.

Thus, a further linear photodetector array, positioned to detect lightemitted from light emitting end 32 of fiber optic 30 would provide asignal indicative of the position of hook 14, and thus item 10 ifdesired, in the "z" axis, that is, in a direction transverse to theplane of the drawing. This signal would be processed in the same manneras signal 26 to provide a three-dimensional determination of therelative position of the conveyor hook and/or suspended item such as itsposition relative to a robot. Alternative means utilizing additionaltargets for providing three dimensional data as to hook location usingmultiple targets are disclosed below.

In the embodiment depicted in FIGS. 1 and 2, the elongate lightconducting means is shown embedded in hook 14. In some instances, aswhere hook 14 is a monolithic cast metal item, it may be more convenientto fix the fiber optic to the hook in some other way as by simply gluingor otherwise adhering to an outer surface thereof. In that event,however, it is preferred to provide a housing for the fiber optic toprevent damage in use. This is readily achieved by providing a groove orslot in a surface of the hook, in which the elongate light conductingmember can be laid and thereafter covered with a protective material,preferably opaque.

Where the invention is utilized to determine the position of a member oflike objects, such as the position of a plurality of identical hooks 14,it is preferred that the position of the light emitting end or ends ofthe elongate light conducting member are in substantially identicalposition on each item. Where this is not practical, or where moreprecise position determination are required, the position of the lightemitting areas of each object may be calibrated.

In the embodiment of FIG. 1, the light conducting members are fixed tohook 14. It will be readily apparent, however, that the members be fixedto suspended item 10. In the latter event, position of suspended item 10is determined directly whereas, in the latter case, it is determinedindirectly by determining the position of hook 10 and by knowing theposition of a suspended item relative to hook 10.

FIGS. 3A and 3B illustrates a basic application of the invention to animportant sector of robot usage, that of taking parts off or placingparts into continuously conveyed containers or transport media. There issubstantial amounts of labor worldwide utilized in this materialhandling procedure. In addition, many assembly operations require ahuman to first take a part, for example, off a conveyor, (for example,an overhead monorail conveyor here illustrated) and assemble it to someother part. He may have to then put the part back on such a conveyor. Inother words, only if the conveyor interaction problem can be solved, canthe assembly process be automated.

A particular embodiment of the invention is shown here utilized toremove transmission clutch parts off an overhead monorail carrier. Inthis particular carrier, 200, there are two parts, 201 and 202, restingin a pocket of the carrier. To keep the cost low, the carriers aretypically made out of angle iron, bent rod etc. and are not overlyprecise in any direction. In addition, they are conveyed often from anoverhead rail 205 and can swing in the direction of motion, side to sideand twist over limited angles. They can vary easily in their positionfrom the reference point of the monorail ±1/2 inch and as time goes on,they degrade still further due to repairs, substitutions etc.

Rather than attempt to build highly precise conveyors, it is of extremeinterest to provide a robot system that can deal with this particulartype of conveyor, not only allowing one to retrofit existing plants, butfurther keeping future conveyor costs low--at the price of certainadditional sophistication in the robot hardware. As can be seen fromthis particular example, however, the embodiment of the inventionprovides a system which can be made at low cost, much less than that ofproviding precision conveyors capable of being used with robots withoutthe invention.

As shown in the top view, a robot 210 is positioned to grab the part offof this particular monorail choosing one of the two parts in thisparticular carrier. At a later time, it may choose any one of parts suchas shafts 220 to 223 located in another carrier 225 onthe same monorail,to be used in assembling with the first parts. For example, robot 350 isused to assemble the parts pulled off the conveyor by robot 210.

The robot 210 can be of any particular type. However, a cartesiancoordinate robot is generally preferred and a specialized one for thispurpose is shown in the drawing. It is noted, however, that polarcoordinate robots can be utilized, although they require much morecontrol sophistication. A polar coordinate robot on moving linear slidesparallel to the conveyor can also be utilized but requires added cost.

As shown, cartesian coordinate robot 210 has an arm 231 moving in andout toward the conveyor line 205 and it moves along an x axis slide 232parallel to the conveyor. The third axis is the vertical axis or z axisout of the plane of the drawing 234.

In this invention, the carrier has been provided in this example withfour targets located, in this example, at the four corners of thecarrier, 240 to 243. These targets can be any type, for example thosedescribed in this application. The use of four such targets is notnecessary, but is preferable in many cases to provide a full 6 axissolution of object position with redundancy. Three targets is sufficientto provide the solution alone. The carrier also contains, desirably inthis case, a snubber rail 245 located beneath the carrier which cancontact certain mechanical guides such as 250 and 251 to restrain theside to side motion. These guides are also shown in the end view (FIG.4). They can optionally be spring loaded as shown such as 251 to keepthe snubber up against the stationary guide 250. A lead-in on thesnubber guides is shown in the top view.

FIGS. 3A, 3B, and 4 illustrated targets such as the four as shownaffixed to the carrier. In this case, these targets provide a signalfrom which the sensing camera unit such as 260 can lock on. This cameraunit can be located either on the robot arm as 261 or external to therobot as 260 (shown mounted by the side but also mounted above thestation). If it is external to the robot, it may also be desirable toalso have targets such as flashing LEDs, 265 on the end of the robot armwhich can also be tracked to provide relative data. Both types 260 and261 can be used to provide data from different points of view.

When the robot sensor such as 260 has locked onto the carrier (or otheritem, see subsequent embodiment) it can then track this conveyor even inits side to side motions in a reliable manner due to the very highsignal to noise ratios of the targets as will be discussed below. Thisis vastly superior to looking at the parts or carriers from their graylevel images in this kind of swinging and uncertain environment. It isnoted that the referenced Pinkney invention or other photogrametricsolutions can offer high resolution data in up to 6 coordinates, x, y,z, roll, pitch and yaw. This is fully sufficient and often more thansufficient to accomplish the job, particularly if constraints such asthe snubber guide rails 250/251 shown are utilized to restrain themotion in one or more axes.

Once the main camera unit such as 260 has locked onto the conveyor andcan compute its position for the feeding to the robot command computer280, then a second system (such as 261), either using a different cameraor simply different lighting, circuits etc. can tell where the part isin the carrier.

An important feature of the invention is the use of one camera unit totrack the targeted conveyor (or other object) while a second subsystem,even with the same camera, senses the part within the conveyor carrieror on it. Such a second camera or subsystem is described in reference 2,and can provide up to 5 axes of data itself (x, y, range, pitch andyaw). This system can be right in the gripper as in FIG. 8 thereferenced copending application (Ser. No. 200,401).

It is noted that the sensing of where the part is in the carrier doesnot necessarily have to be made at the robot station. It could be madeupstream, using for example sensor camera 262. This serves theadditional purpose of signaling the robot system if any out ofspecification situations exist, so as to abort the attempt to grab thepart. This could be a badly mangled carrier, a carrier with no parts atall, a carrier with the wrong part, etc. Thus, identification of thepart obviously can be done as well as sensing its location on thecarrier, pallet, or whatever.

Once the decision has been made as to where the part is and the factthat it's a correct part, the robot moves in to remove it or converselyto place another part back on the carrier. In this case, the controlcomputer 280 of the robot takes the data as to the coordinate positionof the targeted carrier and continually updates the robot's information.With a cartesian coordinate axis, it is particularly easy to make thisapproach since one can simply run parallel to the conveyor (x direction)and only take out the differences in position relative to the parallelline of motion. While this can be done with a polar coordinate system,it is much more difficult to do dynamically. In any case, one needfollow the carrier only in an approximate sensor using the gripper, andpossibly other sensors to take up the difference.

The tracking target approach, for example, using hardware such asdisclosed in Pinkney or a twin stereo approach with two cameras, can beaccomplished using a camera tracking both the gripper and the carrierconveyor (or part) and/or with a camera mounted on the robot arm itself.The reason why this is so successful is that it tracks targets which canhave, and maintain, high visibilities even in an industrial environment.These targets can be differentiated by means of intensity, color orshape. Any and all can be utilized by this system.

For example, in the end view of FIG. 4, light 300 from light source 301behind the carrier can be provided which illuminates the targets 240-243at the four corners (or any other place). These targets can be simpleapertures in plates such as circles, squares, triangles, etc.--whateveris distinctive and can be discerned by a computer vision camera such as260 or 261. A triangle aperture 302 in target plate 241 is shown forillustration.

Alternatively, the targets can be comprised of color filters such as 305and indeed a different color could be used for each different target(240-243) or different part carriers if desired to automatically codewhich is which, if this is a problem, as it could be in certain morerandom applications. In this case, when utilized with white light source300, the color of the target is an immediate indicator. However, in someindustrial environments, maintaining a colored filter may be harder thana simple slot.

It is noted that when utilized as shown, with the light source behind,it may be desirable to put a diffuser such as ground glass 310 (dottedlines) in the slot or near the slot (but not necessarily on the carrier)such that the light is directed over a range of directions. Other moredirectional diffusers such as diffraction gratings, prismatic devicesand the like can also be used where more light is desired at certainangular locations such as the approach path of the robot, or in thedirection of camera 260 for example.

It is also, of course, possible to use the fiber optic based targetssuch as disclosed in FIGS. 1-2 above and in Ser. No. 200,401.

A final type of target of use on a system such as this is aretro-reflective target, such as plastic prismatic retro-reflectors,retro-reflective tape, lectilinear material and the like. This is shownas target 242 in FIG. 4. In this case, a light field 320 must beprovided to illuminate this. If reflectors having a high degree ofretro-reflective capability are utilized, the light source should becoming from the same angle as the sensor (eg. from 260). The sourcecould either be fixed or mounted on the robot. The reflected light fieldis shown 321 directed back along the incident path.

Let us now consider the question of targeting the parts themselves or acontainer of parts such as carton 579 travelling on a pallet in conveyorof FIG. 10 wherein the targets are simply printed on.

There are many means of implementing such targets on parts, althoughthis is obviously somewhat more difficult since one has to consider thefunction of the part and oiten it's aesthetics as well. From the pointof view of the robotic system, however, targets need to be such that atleast 3 of the 4 targets for example are visible in order to provide asatisfactory 6 axis solution to the photogrametric equations. Undercertain circumstances, where there are more constraints, perhaps only 1or 2 targets need be visible.

In continuing the example of FIGS. 3A, 3B, and 4, it is noted that it issometimes desirable to have an auxilliary robot such as 350 (possiblywith polar coordinates as shown) to take parts such as 202' which robot210 has pulled out of the carriers and loaded onto an assembly fixture351. This robot then assembles different parts such as the shafts 220'for example that have also been pulled off and puts the two parts 202'and 220' together and then shoves the completed assembly down on a chuteonto an outfeed conveyor 360.

Alternatively, robot 210 can be utilized also to perform the assemblyoperation particularly if it is provided with rotation about the y axis.A dual robot system, however, is faster since one can be assemblingwhile the other retrieves more parts.

The converse is also true, the previous assembly can be going on whilerobot 210 puts the assembly back on a monorail conveyor of the sametype. For example, in this particular application, a second conveyor canbe located right under the first conveyor 205 on which the assembledparts were placed. This conveyor could be floor mounted or overhead.Robot 210 could also turn around 180° and put a completed assembly on aconveyor parallel to 205.

FIG. 5A

FIG. 5A illustrates a car door targeted according to the invention 352travelling on a conveyor 353. As it passes a light source 355, fourfiber ends 359 are simultaneously illuminated and light eminates fromthe opposite fiber ends 360 to 363 which then form the targets. Thesefiber ends are flush with the door panel which itself may be plasticsuch that the plastic fiber blends with the plastic door. Indeed, thefibers carrying the light 370 to 373 may be cast right into the plasticin the injection mold. They may be in the door panel sheet, laid in justas if they were regular glass fibers in a SMC (fiberglass) door, ormaybe carried in or adjacent the ribs of the door (if any).Alternatively, where there's an inner and outer panel, the fibers may beplaced in between.

A camera unit 368 looks at the light from the fiber ends 360-363. Thiscamera may be located overhead the conveyor and/or on a robot arm comingin to pick the door up for example.

The light source itself may be pulsed to create a higher signal to noiseratio of the targets relative to ambient light. It is noted that each ofthe fibers comprising the targets may transmit colors to allow colordiscrimination on that score. Indeed, one might even think of the fibers370 to 373 as bundles of fibers. Indeed different numbers orarrangements of fibers and different arrangements of the target ends 360to 363 could be used such that varying codes were used to deliniatewhich was which and what door type etc.

It is noted that paint, ink or other film or coating type targets can besprayed on parts utilizing spray marking guns. Particularly effectivefor this in terms of producing nicely shaped targets such as trianglesand other items that have a very recognizable shape is the DiffractoTurboJet, U.S. Pat. No. 4,269,874. Clearly, a marking station on thepanel for example could be utilized to spray certain targets on withpaint. These targets can be provided with special paint such asphosfluorescent, infra red absorbing or ultra violet reflecting,something that would tend to distinguish them under certain types oflighting. They could even be clear coating which would be invisible tothe eye but under certain lightings would fluores or absorb certainwavelengths preferentially. The latter would be particularly nice forfinished parts. However, any sort of paint could be utilized for exampleon an unpainted door which would later be painted just as long as it waschemically compatible or would be removed in a normal preparationprocess.

FIG. 5B illustrates the color filter plate 380 which can be placed infront of the group 359 of four fiber ends of FIG. 5A. This color filterplate has filters red, yellow, green, blue as shown which cause thelight eminating from each of the fiber ends 360-363 to show those colorsfor example. These colors could all be infra red or any other colors.

An alternative for differentiating which target is which, is to actuallyuse different light sources for each of the fibers and modulate them atdifferent rates or pulse light on each of the fibers sequentially.

FIG. 6 illustrates another application of the invention, this one beingto a windshield 400 the edges of which as well as other features areviewed by cameras 405 and 406 having an included angle, 0 between themsuch that `stereo` vision in terms of depth perception is provided.

This stereo vision or any other vision of this part is made vastlyeasier by the deliniation of the object edge provided by the `lossy`fiber 411 which runs around the periphery of the part eminating or`leaking` light at each point as it goes when illuminated at its end bylight source 410.

While the whole edge has been deliniated in this particular example, itis obvious that only sections of the edges of a part such as this arerequired for accurate placement of it. It should be noted as well thatit allows for actual mensuration of the part itself since the contour ofthe windshield edge is desired to insure that it will fit into thewindshield opening of the car in a correct manner.

The fiber in this case can be cast into the windshield glass at the timeof manufacture and could even be a glass fiber itself simply of slightlydifferent characteristics. Alternatively, this fiber could be reallyjust a portion of the same glass made in such a way as to convey lightaround the windshield periphery.

It's obvious that these same principles could be utilized on plastic orfor that matter, even on metal parts where the fiber would simply beglued onto the periphery of the part or be covered over only in part bythe metal if desired.

It should be noted again that the light eminating from the fiber can beinfra red or any other wavelength that might be desirable for betterdeliniation of the surface. Indeed, the fiber in the case of plastic,can be buried such that it could not normally be seen under visiblelight but that infra red radiation, in this case eminating from thefiber outward through the glass or plastic could be seen.

Clearly, more than the four targets shown in FIG. 5A on a door can beused as can more zones than simply the periphery of the part of thewindshield such as FIG. 6. However, these are the two principleexamples, namely four points is the sufficient solution including aredundant check of the 6 axis, photogrametric equations, and of coursethe periphery is the main item of interest when looking with stereocameras.

Note that in FIG. 6, many fibers are lossy just by themselves. Ifadditional losses were required, for example when they were imbedded ina matrix, the fibers could be roughened for example.

Clearly, plastic pallets and carriers can also be so instrumented.Indeed, light conductive paths can even be built into the plastic of acarrier such as FIG. 4.

FIGS. 7A and 7B illustrates another method for targeting objects. Inthis embodiment of the invention, a cylinder head 500 is targeted, inthis case on the rocker cover rail 501, with targets 502, 505, 506 and507 on the four corners of the rail perimeter.

Target 502 in this case is formed by a single depression in the rail,for example, the conical surface made by a drill as it just touches themetal. Such a conical surface reflects light at diverse angles from thatof the flat machined face of the cover rail itself, and therefore makesan excellent contrast target when viewed at appropriate angles. Forexample, when light from source 510 carried on robot 512 hits the coverrail containing such conical target, light is directed back at camera511 from the target because the face of the target is pointed more orless at the camera 511. However, the light from the cover rail isdirected off at a angle away from the camera. Therefore, it appearsbright against the background of the cover rail and against that of thecast surface of the part. In other cases, the rail surface would bebright and the conical surface dark. The fact it is a cone, means thatapproach from any direction in the plane of the rail face desirablyproduces similar results.

Shown in the top view for illustration, different sets of targets havebeen shown on each of the corners. However, it is considered likely thatin any one case, one type of target would be used. For example, thetarget cluster 505 contains four such conical faces or for that matter,four targets of any sort such as will be shown in FIG. 8 for example. Inthis case, four targets of course are much more unusual than a singlepoint and would be unmistakable relative to any sort of backgroundssince nothing else on the part could have such a cluster reflection. Inthis case, the center of the four dots gives the center of the target.

The same holds true of 506 which is a 3 pointed version, also providinga center. Cluster 507 having two points, while probably unmistakable,does not have a center point exept in the one plane. In this case,therefore, the center of the dots themselves would have to provide theanswer in one plane.

FIGS. 8A and 8C illustrates another target method, also applied in thiscase to cylinder heads but of course general to any sort of part. Inthis particular case, the part is cast where it is shown that oncylinder head casting 530, there are appendiges cast which do notinterfere with either the assembly or the function of the part. Theseare shown as 531, 532, 533 and 534. These appendiges are indeed targetsand of course are unmistakable in any sort of view.

To make them more unmistakable, certain angles have been cast into theirsides. Such as shown in the end view of 532, the particular angle ofreflection transverse to the head axis is such that when overhead lightfrom an overhead light field 535 is projected, that these facets shootlight off to a camera at a preferred angle. In this particular case, theopposite one 531, would have to be made in a different way as shown,such that it too had a facet in that direction. It is noted that thedirection can be chosen such that no other features on the part havereflective angles in the same direction. In other words, for any part,no matter what it is, one should be able to find certain angles at whichtarget data can be made to show up either brighter or darker than therest of the part with no other, or at least a minimum of other partfeatures having angles at these directions. This of course helps thediscrimination in a passive manner.

Similarly, certain targets can be bevelled in more than one plane, suchas 533 as shown such that when viewed from either of two angles, abrigher reflection is shown. (Conversely the lighting can be at an angleand the camera located overhead.)

Also shown in this drawing, are cast in target cones or crosses such aswere applied in a separate drilling operation in the FIGS. 7A and 7Bversion above. These are shown as 540, 541, 542 and 543.

In this case, a male portion in the mold itself provides a suitableindentation in the part. Since the targets can be so cast, they can beof many other shapes besides simply conical surfaces, holes or othereasily machined shapes. For example, cross shapes like the facets of aPhillips screw, which are undeniably discriminate as targets as opposedto other features on the object which can have some resemblence toconical shapes.

It should be noted as shown, that such shapes do not necessarily have tobe indented in the part but can be raised such as the equivalent feature545 shown sticking up. Such a knob or bump on the part, however, can bein the way of the function of the part and its handling if it is notproperly positioned. It is therefore thought since many parts generallyhave flat surfaces, which are either functional or to simplify handling,that the best means is an indentation in those surfaces which does notinterfere with either purpose.

As shown in 8B, a cast or drilled in cone, cross etc. in a surface 545into the surface of material 546, can optionally have a transparentplastic filler material such as 550 placed into it to cover up a portionor all of the depression or even provide a raised portion sticking outfrom the surface of 546.

This plastic material can serve several purposes. One purpose is that itsimply protects the surface of the depression from rust anddeterioration. This could be quite important in let us say a brightshiny drilled portion on an aluminum or steel part which in timetarnishes or rusts.

A second purpose is that the filler itself makes a different opticalelement out of the mirror formed by the cone surface (cross etc.) and inthis case forms it into a prism which can have use for either spreadingthe light or directionalizing it.

A third potential reason is that the plastic filler may itself be chosenso as to preferentially reflect light only of certain colors. This thenallows another form of discrimination of the targets based on color.

The principal disadvantage of using such a filler is that a separateoperation must be made to put the plastic in, which cannot normally bedone easily on a machining line. One exception, however, is the processwhereby first the reflective hole is drilled into the casting, thenplastic is sprayed into the holes very simply, and then a finalmachining pass required for other purposes is done which in the processshaves the excess material away leaving the plastic flush with the holesurface. This obviously then only adds the spray guns to the process.

FIG. 9 illustrates another example of targeting, this time on a plasticbody panel 570 illuminated by light field 571 which is then directed bya target molded into the plastic surface 572 onto camera 575. As shown,the target is reflective and is composed of a diffraction grating whichdirects particular colors or, in general, light of all colors, at anglesfrom the surface. An alternative is that the target 572 be composed of amulti layer interference elements preferably in plastic which also candirect light at preferential ang1es in preferential color combinations.

If the camera 575 is capable, as is a color TV camera, of color sensingas well as spot shape sensing, it can then differentiate these colorsand unmistakably identify that such a color spread can come only fromsuch a target. This can be done even in the presence of strongbackground, as from the surface of object 570. Such color combinationscan also be coded into the targets to identify the part 570, its angleof orientation etc.

Rather than mold the plastic into the part, it can also be simply gluedonto the surface of the part (578). If a thin reflector film, such as578, it may, even though sticking up, be non-interfering with thefunction of the part. However, for plastic outer body panels in cars,the flush target mounting such as 572 is preferred. These targets onpainting of the car, become covered over. For example, if the targetsare on the door panels of the car, which are mounted to the car at thetime of painting, their presence is lost once the car is painted. Thetargets used for such mounting for example, should be flush and createno disturbance with the panel surface once they are painted. It is alsonoted that targets can be built into objects however to actually be partof the object's appearance. The necessity of covering the target updepends greatly on the aesthetic characteristics of the object.

Another possibility is to utilize targets which are seen as targets onlyunder special illumination which would not normally be present in ahuman situation. For example, consider target 572 which could be eithermolded into the panel or for that matter, simply a portion of theplastic surface of the panel itself treated with a special ultra violetflorescent material. Only under ultra violet light would this targetportion of the panel actually be visible relative to its surroundings.

This is partly true of the multi layer diffraction case above where theline spacing of the diffraction pattern or the multi layer material andspacing could be chosen such that only under certain colors ofillumination, and then perhaps only at certain angles, could the lightbe strongly seen relative to the surroundings. This then would beparticularly easy to arrange if such wavelengths were in the UV or infrared just beyond human vision. The near-IR is an exellent region forsensing with present day solid state cameras for example.

It should be noted that such targets do not necessarily have to bemolded in but could be evaporated onto the surface such that the raisedamount of material is virtually negligible. Naturally, in crudeapplications, such applied targets such as 578 could simply be whitecrosses of plastic glued on. Clearly, this could be unobjectionable inthe final product if in the final painting process there is a wash thatsimply removes the glue and target. One advantage of the fiber typesshown in FIG. 5A, is that the fiber end can be extremely bright andflush to the part surface.

FIG. 10 illustrates another application of the invention, to tracking acarton such as 579 travelling on a `car` or carrier conveyor such as590. The carton has been randomly placed onto the carrier and it isdesired using programmable robotic means to grab the carton and pull itoff at a certain station.

To accomplish this, all sides of the carton have printed on target setsof which 580, 581 and 582 are visible in the drawing. These target setscan be of any usable description and remain with the carton always. Thebeauty of this is that they can be utilized for tracking in themanufacturing plant, and for robotic warehousing purposes and throughoutthe distribution chain, even for example in a supermarket providerobotic unpacking of the product and place the product on the shelves.Naturally the product packages within the carton, such as let us say eggcartons, or milk cartons, cans etc. can also be targeted for the samepurposes since all those have printed on labels or the like.

While each face in this case is shown with four dot type targets, thesecould clearly be of any number or type. The carton also could be codedto indicate the goods within the carton. In the extreme case, this wouldrequire a UPC type codes (eg. 591), and indeed a miniature UPC codeitself could constitute one or more targets. However, it is consideredthat most of these would use much less complicated codes since therewould normally be no need for such large amounts of information.

It may well be necessary to code the object or targets since alldifferent boxes would have different target spacings due to their ownshape and one would first wish to decode which type it was so that thespacing of the targets could be known to the computer of the robotmechanism and fed into the calculations for the various solution of thephotogrammetric equations.

For example, it could well be that a special code target might also beused such as 591 which would include all the photogrammetric solutiondata for that carton plus an indication of what was inside if desired.The robot camera system could read the code first and from samedetermine the various target location data on each of the faces of thecarton including the target shape and size, the target spacing, how manytargets there were and for example the shape of the product itself,whether in a square carton or what have you.

FIG. 11

FIG. 11 illustrates a similar concept this time using targeted toolssuch as the grinder 600 driven pneumatically via air hose 601. It isdesired that a robot with a gripper come pick this grinder up and dowork on an object, for example the leaded-in zones of a car body at thepanel junctions.

For this purpose, the tool gripping area 610 itself is targeted, in thiscase using light emitting diode targets 605, 606, 607 and 608, also fedthrough the embilical 601. These diodes can either be on continuously,may be flashed to provide has high signal to noise, or rippled such thatonly one is on at any given time. The latter is useful if photo sensorsresponsive only to one point at a time are used such as continuous spotdetectors (eg. UDT SC-10).

Naturally, rather than light emitting diodes at the tool, fibers can beutilized to feed this data to the same points from one or more remotelight sources.

The robot hand with camera would approach this tool and via the targetsgrab the tool at the desired location, in this case, the cylindricalsurface 610 which would be grabbed by `V` shaped grippers. It is notedthat the targets can be placed specifically so they bracket this areaand this is a preferred mode of target placement in such instances.However, this is not necessarily required. The targets could be placedsuch that gripping would be known to the computer to occur in any otherlocation as well.

In this case, it may also be desirable to have a code such as 620 shown.This code could carry with it the data of where the part is to begripped, whether it's between the targets or somewhere else, and again,what tool it was and perhaps other data as well.

It is noted that a target such as 532 and 531 of FIGS. 8A and 8C, ifthey project from the object in one or more planes, can allow moreaccurate solutions of the various pitch and yaw data which are derivedfrom the projected spacing of such targets viewed by the camera.However, the more the target projects from the part in question however,the more the possibility it is objectionable for handling or aestheticreasons.

FIG. 12

FIG. 12 illustrates another application of the invention to the assemblyof car bodies. In this case, it is desired to assemble a deck lid 660onto the body opening formed by two fenders 661 and 662 and the otherportions of the body not shown for clarity. This problem is very similarto that of fitting the doors in a door opening or the hood in the hoodopening and is optimally improved using optical sensing as disclosed.

As shown, a robot arm 650 carries with it a tooling fixture 651containing vacuum cup fixture 652 and 653 which attach to the deck lid660 to be put on. The fixture itself contains optical sensors, in thiscase 670 and 671 which are used in the mode shown here tracking targetsas well as to measure certain variables of the part itself usingconcepts shown in the referenced applications.

Such applicable sensors are shown in references 2, 12 and others.

As the robot approaches the car body containing the fenders carryingwith it the deck lid 660, sensors 670 and 671, which contain linear ormatrix camera units, have determined the position of the deck lidrelative to the cameras themselves. In other words, the cups 652 and 653can pick up this deck lid in a relatively random fashion from let us saya roller conveyor and have the cameras compensate the robot for thisrandom position by sensing the edges of the deck lid. Alternatively, thesensors can sense the deck lid edges ideally and cause the pickup to bemade in the correction location. It is likely too that other cameraswould be located on the other sides of the part, for example, as shownas 680 (dotted lines).

When the robot is relatively far away from the body, the camera unitwhich also contains for example illumination source 675, picks up thereflected image of a stamped in cone targets such as 665 into thefender. Alternatively, for even better contrast, a hole 665 can beprovided, back illuminated if possible by a light source such as 666.Unfortunately, however, in most portions of panels, extraneous holes arenot desired. Such stamped in targets are however extremely possible andcan be accomplished just as in the case of the cast in targets of FIG.8A and 8C in an analogous manner. The fiber based targeting systems areideal if they can be employed economically.

As the sensing of targets such as 665, and 667 on the opposite side, aswell as other targets around the rest of the periphery of the deckopening, allow the robot system to home in on the body. Note that unlikeprevious embodiments, it is not a single camera which is seeing alltargets, but the ensemble of two or more cameras whose combined targetdata gives the position and orientation to the part. As the camerasensor unit comes in for its final approach, an oblique light projectorunit such as 672 illuminates the portion of the part itself from which atriangulation data as to the exact range to the fender 661 can beprovided at a higher resolution. Such a sensor unit incorporating thishas been shown in reference 2 and other references.

As the part then fits into the opening `D`, the gap width `W` on eachside is sensed by each of the cameras on the four sides and optimizedfor the car body in question. When the deck is optimally positioned,then various hinge screws and bolts are run down to lock it into place.This process therefore not only generates a fully automated deck (ordoor) placement, but also creates a optimal body fit for highest qualityperformance. This operation does not necessarily require the use of thetargets and can be done in a targetless fashion particularly if the bodyis stopped when this is occuring. If the body is however in motion, thetarget data definitely is very much desirable such that its side to sideand forward/backward oscillations can be tracked on the approach.

It may also be required that two sets of camera magnifications be used,one at high magnification to determine the distance `w` and one at lowermagnification to track the targets. This depends on the application andnaturally is not as desirable as just a single unit. Further, in thiscase the targets are shown being covered up by the panel, in otherwords, they are out of sight in terms of the body itself. This can betrue in both doors, decks and so on. Some of the tracking however couldbe done by targets which were visible on other portions of the body andnot covered. This would allow the targets to be tracked even at the timeof actual panel insertion and bolting which would be desirable on movingparts. For this purpose, it is thought special targets should be stuckonto the body such as target 681 shown which has been stuck onto thefender and is, for example, comprised by a white background with a crosson it. Such targets might be viewed by a completely separate camerasystem mounted to the side or overhead or on the robot arm 650 itselfrather than on the tooling.

It should be noted that in any of the above embodiments, targets shouldbe as distinct as possible. If possible, certain types of reflectivetarget material such as plastic retro reflectors and retro reflectivetapes can be of extreme interest as long as they can be placed on theobject in a manner that does not ruin its function. Such tapes andtargets, therefore, are best suited for use on objects which do not havean aesthetic purpose and some of these would certainly be all conveyorparts, cartons etc. The problem, however, with these targets is thatthey are generally of materials which must be attached and this cancause difficulties in terms of both the cost of attaching the targets inan accurate manner (remembering that for best operation, with multitarget systems, the target spacing and orientation needs to be known,such that the photogrammetric calculations can be accurately solved. Thesecond problem with these materials is that they are often plastic andin some cases, plastic will not survive the remainder of the processwhether it be hot washes, heat treat, or what have you.

It should also be noted that targets, when applied can be removed foruse on subsequent parts. For example, retro reflective glass targets ofvery high contrast can be screwed into tapped holes on the part at verywell known locations and screwed off later on and used again. This wouldbe easily accomplished for example, on the cylinder head of FIGS. 7A and7B if the tapped rocker cover rail holes for the rocker cover wereutilized to carry the targets which were screwed into those holes,preferably automatically. At the final rocker cover installation, thesescrews would be taken out and the rocker cover bolts put in. Naturally,however, this adds two operations, screwing in, and screwing out, to theprocess but it does utilize the same holes that are put into the partanyway. Other targets could be attached with glues etc. which could betaken off the part with solvent and off the target such that it could bereused again after cleaning. This is discussed relative to FIG. 13.

In addition to the above ideas, ther are several other continuationsfrom the previous application that should be noticed. For example, FIG.8 of Ser. No. 200,401 discloses instrumented grippers with fiber opticsensor units including a triangulation projection source to which allows3 axes of data to be obtained. It is noted herein that up to 5 axes ofsuch data can be obtained using projection of multiple beams or fourbeams to get four or five axes of data. This allows the pitch and theyaw of the part to be obtained as well as the range, plus the xy imageand is further described relative to FIG. 15 below.

It is noted that the robot arm may be instrumented for guidance withsuch a sensor either using LED or diode lasers as targets or via fibers.Such concepts of guiding robots with external cameras using targetedarms has been shown in the copending application of reference 13.

Color discrimination of the various targets can be made by using colorTV cameras or with simply a color sensor in front of a detector ifapplicable. For example, relative to background levels, if all targetsare infra red emitting such as infra red LEDs, then an infra red, bandpass filter can be placed in front of the camera such that greatlydiscriminates against the white light image background and showsprimarily the infra red targets superposed thereon.

Furthermore, the holes put onto cylinder head in FIGS. 7A and 7B can bemore than just conical, they can be actually be drilled in deeper suchthat they tend to absorb all light. In this case, one would look at theangle of reflection from the bright machined face of the rocker coverrail and the target holes would show dark.

It should be noted in FIG. 8B, a blob of plastic or a blob of siliconecould be put on top of the part to act as a target. Indeed, if a linearstrip of silicone for example were utilized, this would approximate thefiber arrangements shown in FIG. 5A or 6 and indeed light can betransmitted through and around the part illuminating edges of itthereby.

FIG. 13

FIG. 13 illustrates one example of a reusable target, in this case, aspecial screw 700 which is screwed into a threaded hole 701 in a part702 such as the cylinder head of FIG. 8, engine block, or for thatmatter just about any machined part that has a threaded hole. Thesethreaded holes would as has been pointed out, be almost certainly holesthat already exist on the part for other purposes and as for assemblywith the target part taking the place of the regular part up until thepoint of final assembly when it would be removed.

The target screw is built like a socketed cap screw but instead of thesocket hole, in this case, being at least partly filled with aretroreflective target 705, which is ideally comprised of plastic orglass retroreflective material for example that commonly used onautomotive reflectors or specialized types built by 3M and othercompanies.

If desired, a color filter such as 710 can be utilized on top of thisscrew or as part of the retroreflector to give a preferential colorsignal from this particular bolt or stud if it is desired to distinguishit against others. The reflector design itself may also provide suchdistinction being multi-pointed or what have you.

This particular arrangement provides an extremely high targetdeliniation, and allows the targets to stand outward from the partsurface if desired (as for better photogrametric solution purposes) bysimply having a long threaded length. Furthermore, this stud is arelatively low cost item and can use automatic lines to put in and takeout. The only disadvantage of course is that it must go into a hole thatis later used which means in the final assembly process, the targetcannot be used unless the part is not moved during assembly after thetarget is taken out.

While a screw type has been shown, it is clear that other arrangementssuch as bayonet, snap in/snap out, or other targets could be utilizedwhich could be removed with special tools from otherwise clear holeswhich later would accept trim strips, rivets or what have you.

In other cases, the target itself might simply have a pointed end suchas a pin which could be stuck into the object material and later removedleaving a hole which would cover itself over if the material wasrelatively compliant. This could include, for example, seat materials ormeat on overhead conveyor lines where the carcass itself could havetargets put in it.

FIG. 14

FIG. 14 illustrates an application of the invention to working on acontinuously moving car body assembly 780. In this case, a roboticsystem according to the invention is provided complete with camerasystem 785 which locks onto the body targeted with reflective targets781-784 in the working region causing the robot to track the motion ofthe car body side to side, backward and forward on the body "truck" (notshown).

The sensor unit 785, in conjunction with robot control computer 789,controls the robot arm 800 to move an abrasive belt grinder 801 to grindout the lead 790 fill-in between the sail panel 791 and the roof panel792. There are two forms of additional optical sensor units of use inthis embodiment. The first is 805, such as FIG. 16 of the referencedapplication which allows the attitude of the belt grinder to the surfaceof the body to be determined for tracking purposes. The second (notshown) is a contouring sensor such as FIG. 4F of Ref. 12 which contoursthe leaded zone of the body to feed back contour coordinates to thegrinder and update the amount of metal left on and judge whether or notfurther grinding should occur and if so, from what angle (determined inconjunction with the dynamic tracking data at low resolution from thetarget sensor 785, and at high resolution from the on-board sensor 805).

Utilizing all three of the optical separate sensor systems plus forcefeedback, a complete grinding cell so to speak can operable on-the-fly.If the car can be stopped in its motion, the target based system is notas much required for tracking the gross motions of the body and theother two sensor systems are sufficient. However, the target system is agood "insurance" for rapid approach.

In the above application, considerable amounts of specialized hardwareare of use, much of which has been discussed in the referencedapplications. For example, camera units are best provided by solid statematrix arrays such as the GE TN2500 and the new solid state TV colorarrays now appearing on the market by Sony and others.

In terms of light sources, flashed Xenon light sources are very good forilluminating targets with brilliant high signal to noise pulses, evenwhen color filters are applied. Also, such flashes do not cause thesolid state cameras to bloom, a desirable advantage.

Desirable laser light sources include diode lasers operating in theinfra red made by RCA and Laser Diode Laboratories. Of interest too isthe Mitsubishi 4001 laser diode which is partially visible.

The high powered infra-red LEDs such as the Texas Instrument types canalso be utilized for such illumination through fibers or what have you.LEDs are very convenient in that they are low power consumption and canbe modulated as can the current range of diode lasers.

The approach described relative to FIG. 5A and 6 holds for all kinds ofother parts such as tires, parts of aircraft, furniture, just about anypart where some sort of method of casting, molding or otherwise placingfibers into the part can be done. Even metal parts can have integralfibers if they can stand the melting temperature (eg. quartz fibers).

It should also be noted that the part does not necessarily have to havefibers cast or molded in. One can also have a fiber placed onto thispart, for example, glued to the part around its periphery or at specificpoints. These are then illuminated and can then be used for the samerobotic and other purposes as shown above.

This gluing operation, however, generally requires additional labor,either human or robotic, although it could be done on an automaticin-line machine as well.

It should be noted that while fiber optics have been discussed as thelight carrying medium, it is clear that a transparent silicon bead laiddown on a part is also light transmitting although less so. Thisparticular use of fibers and other light transmitting mediums appliedinto or onto parts is particularly appealing for many applications wherethey are to be substantially robotically handled and, therefore, wherethe cost of applying the fibers in and illuminating them at differentstations is made up by savings due to reduced complexity of roboticautomation utilized.

The application of such concepts to things such as tooling was discussedin my recent copending application (ref 2) on robotic castinginspection, where sensors were in the tools to sense part condition.This disclosure has expanded on this to provide fiber illumination oftool location to allow handling or size determination of tools. This isalso related to a copending application entitled "Method and Apparatusfor Detecting Wear or Breakage in Tools". Suffice it to say that toolscan also be illuminated like the J-hook of FIG. 1, to provide meaningfulindicators or targets to allow pick up by robots or other automation.One can consider such tools as cutting tools, small drills, routers,pneumatic wrenches, saws, lasers, weld heads etc. All can beinstrumented in this manner. Even small things such as sockets forwrenches can be so instrumented.

Note that `light` in this application refers to all wavelengths ofelectro-magnetic radiation IR through UV.

Similar fiber optic emitter targets can be the grippers or arm robotsthemselves, replacing LEDs or other types on the grippers such as shownin copending application entitled "Electro-Optical System for Control ofRobots, Manipulator Arms and Coordinate Measurement Machines".

Suitable fibers include, at the low end of cost, the Dupont Corfonplastic fibers as well as glass fibers made by American Optical, Corningand numerous other manufacturers.

It should be noted that image transmissive bundles can be utilized toremote the images of sensors shown in this application as has been shownin the referenced copending application which this application is acontinuation in part. Such fiber optic bundles are made by Nippon SheetGlass, Olympus and others and can have very high resolution.

It is noted that image scanning photo detector camera arrays and solitstate TV (matrix array) cameras, while preferred, are not the only meansof viewing the targets of this invention. Other TV cameras can be used,as can in some incidences scanning laser beams or even fixed detectorsoptimized for a preferred target signature. Continuous or quadrantposition detectors (such as UDT SC-10's) can be used as well todetermine the image position of a single spot or target at a time.

Shown in FIG. 15 is a sensor according to the invention providing animprovement on some of the fiber optically based sensors of theco-pending application Ser. No. 200,401. This particular sensor shown isa multi range sensor of unique small size according to the inventionwhich in this case is shown being so small that it can be built into thegrippers of robots. It does not require targeted objects, but can becombined with other embodiments to work in conjunction with targets aswell.

As shown sensor 900, located in this case in one half of the gripperportion 901 of a robot end effector is comprised of light sources 905,906 and 907. (In this example there are 3 light sources although therecould be any number.) These light sources are diode lasers or in manydesirable instances, they are 0.005 optical fibers remotely connected todiode lasers with only the fibers brought to the sensor.

In any case, light from each of the fibers is focussed by single lens910. However, due to the variation of positioning of the fibers, thelight is focussed at different distances and at different anglesdepending on the position of the fiber. This is ideal for providing amulti range, multi resolution sensor, with highest included angle andresolution at the shortest ranges as is desired for accurate partpick-up and other purposes.

Light source 906 is focussed at the nominal range to the part 911 shownin the drawing forming a reflected spot 912. This spot is, as has beendescribed in many copending applications, imaged by lens 915 onto anintegral photo detector array 916 (dotted lines). However, in this case,again for compactness, the image is formed onto a coherent fiber opticbundle 918 and carried to a remoted matrix photo diode array. Thus, inthis example, all light sources and sensing can be done over fibers ifdesired. This is attractive for thermal and electrical isolationpurposes, plus light weight on small robots.

A suitable window, 920 is provided in front of the sensor housing.

The other two light sources, 905 and 907, on either side of the nominal,focus at different distances and at different angles. The larger theincluded angle θ, the more the resolution. Therefore, it can be seen asthe image forming capability associated with 907 is at the highestresolution with the part closest and this is used for the fine approachof the sensor where the range `H` might be only half an inch. In thecase of 905, `H` might be set up for 10 inches.

Obviously, this sort of an arrangement is fine for maintaining areasonable focus of light sources at different ranges. However, with asingle lens 915 one needs a narrow aperture to give large depth of fieldand maintain the spots projected in reasonable focus over a wide rangeof object locations. Alternatively, a zoom lens 915 can be used tomaintain focus over the range.

Since spot centroids are being measured, it is noted the spot image canbe somewhat out of focus and still be utilized (see reference 2 or 4 forsuitable circuit processing). Optional white light sources can also beused with this arrangement to provide a edge image lighting with thepart 911.

It is further noted relative to FIG. 15 that each diode laser or fibercould be focused by an individual lens. While more complicated, thisallows more angular spread between beams. It is contemplated that onlyone beam would be turned on at once, suitable for the range in question.However, even if more than one were on simultaneously only one isgenerally in the field of view of lens 910 at a time. If there are twoin the field, they can be discerned from their location. Indeed, twodivergent beams can be projected on purpose at once in the field, one todetermine range and the other to give angular orientation from the beamseparation on the target, knowing range.

It is further noted that this invention is very useful to controlrobotically positioned non-contact processes such as laser welding,drilling etc. especially on continuous lines. In terms of processes ingeneral, the invention applies to welding, drilling, grinding, cutting,hardening, and any other material removal, addition on transformationprocess.

The characteristics of targets used in this invention generally includedistinctive shape, light reflection, light transmission or lightemission characteristics relative to the normal surface of the objecttargeted. Where the `normal` object has targets, a better definition isrelative to the rest of the object surface, i.e. the untargetedremaining portion. Light emission, reflection or transmission can bedistinctive in color, direction, distribution of direction or color,shape, and intensity.

In the case of the fiber version and other active targets, the targetscan also be diverse in their light modulation frequency.

It is noted in the application of the invention to practical plantproblems, that photodetector arrays are much prefered over the analogtube based TV cameras used by Pinkney and other photogrammatists.Particularly photodiode arrays such as the GE TV2500 do not requirefrequent calibration and therefore can be relyed on much more to giveaccurate dimensional data as to target or spot location. For example, aTV tube drift of 3% in the apparatus of Pinkney et al can create agenerally intolerable error of 0.3" at 10" standoff in the range dataalone. The arrays used in this invention preclude this possibility.

It is further noted that in the embodiments shown herein relative tocontinuous conveyors, if conveyor speed is known, the trackingrequirements are reduced accordingly.

It is also noted that the snubber rails 250 and 251 are but one exampleof means to constrain motion or velocity in one or more axes of anobject in this invention. It may also be useful to constrain velocityfor example using electro magnetic or viscous fluid damping. Constraintsof this sort generally make the total robotic handling or parts workingsystem easier to control.

This disclosure has described many ways of adding targets to objects.Other ways of making the target part of the object have also beendescribed. Where the object is one which is in it final form and locatedin a position that it can be seen by a consumer who expects it toprovide a pleasing appearance, there is considerable requirement to makethe targets used in the invention either essentially invisible oralternatively make them have asthetic value of their own.

For example, a doped target zone of a plastic dashboard piece canfluoresce under UV light but remain invisible in normal illumination.

Alternatively, a portion of the object may contain a special dopant tocuase it to reflect or absorb in the IR more than normal.

A desirable condition exists if one can make the targets part of theoverall design to provide for example a pleasing accent feature whichare viewed as part of the design. Where possible the targets can also befunctional features such as holes, knobs etc.

For multi-target application involving 3, 4, or more targets (to providemaximum solution capability of the photogrammetric equations) it isnoted that the targets do not have to be equi distant or otherwise orrigidly arrayed in their relation. Thus a variety of astheticpossibilities exist. For example:

On rectangular parts, the 4 corners are desirable where targets can besquares, circles or other shapes.

On circular parts or sections thereof, the 4 targets 90° are desirableor 3 targets 120°.

On irregular parts, the targets can be in any logical arrangement.

Examples are:

a furniture leg 950 with round head tacks 951-955 at corners serving astargets (FIG. 16A)

automobile grills 960 with decorative square target fascets 961-965 orslots near the corners (FIG. 16B)

automobile steering wheels 970 with triangular bright inserts 971-973 inouter edge of three 120° spokes (FIG. 16C)

It is also noted that within the target slots 241 etc. of FIG. 4,transmissive diffraction gratings and other preferential diffusers oftransmitted light can be located.

It is further noted that light from the fiber end 20 for example neednot necessarily be imaged by lens 40, but can be detected directly byone or more detectors.

It is noted that in many plant applications where the targeted objectgoes through many processes or are repetitively used (eg. the conveyorcarriers of FIG. 3), that the targets can degrade or be knocked off,destroyed etc. Thus it is desirable in many cases to have a superfluousnumber of targets.

Since only three targets are needed for a complete photogrammetric 6axis solution (and even two will do if certain aforementionedconstraints are used), 2-3 is then the base number of targets per objectgenerally desirable. In many cases, a fourth target is desirable toprovide a redundant solution however.

This invention therefore considers the purposeful addition of extratargets also in known locations relative to the 2-4 basic targets andfor the additional step of determining which targets are present andusing an optimal solution for those.

Consider FIG. 17. Illustrated is a conveyor pallet 980 used repeatedlyin a plant equipped with targets 981-984 and extra redundant targets986-988. In one mode of operation, normally camera 990 and computer 991are programmed top consider only targets 981-985. If, however, one ofthese targets is missing, the images of target-986, 987 or 988 isutilized. In general the rationale is to use the remaining targetclosest to the missing one but the best rationale is to use whatevertarget combination gives the best solution (ie. most accurate) of thephotogrammetric equations.

It is contemplated that certain additional targets might also be appliedto provide, in certain instances, more accurate solutions for example,to pitch or yaw variables in the plane perpendicular to the lens axis.For example, one might choose at certain stations in the line where ahigher degree of sensing in one or two of the variables were desired, touse target 988 on purpose, instead of 985 say.

Naturally, if all targets but 3 are damaged, one uses the remaining 3regardless. However, the invention can include the additional step ofsignaling a control that pallet SN 1368 say is down to its last 4targets and should be repaired.

It is further noted that a system input or verification station is oftendesirable in a system such as shown in FIG. 3.

For example, consider providing sensor 262 at a position where theconveyor carrier is well positioned such that the targets can be checkedfor presence and their locations verified if desired. Not only is thishelpful in keeping the system in control, but if each carrier wasserialized or sequenced the actual target location can be measured atthis station and the locations stored in a computer, such as 280,relative to the carrier in question. This allows different carriers ofdifferent parts in different states of repair to be intermixed on thesame line with no loss of target position accuracy. This is importantsince the accuracy of the solution of photogrammetric equations (used toguide the robots on the line such as at the station of FIG. 3) is basedon the degree of accuracy with which the relative location of thetargets to themselves and to the carrier body is known.

Only one such verification station (which could also have a carrierserial code reader as could the station of FIG. 3) is required per line.It also, as has been mentioned, helps monitor damaged carriers anddamaged carriers could then be automatically routed off-line for repair.

Note that verification station can also be used for parts when they arein a fixtured or otherwise known correct position at some point in aline. Any missing targets as well as locations can be verified beforethey enter the system.

This invention will be useful in a wide range of applications and it isenvisioned that a standard sensor computer unit can be built essentiallyindependent of the application. The user would only need type in orotherwise enter data into the computer such as 280, to tell the systemthe pertinent target location and spacings on the parts or objects to behandled, assembled, or worked. Thus it can be reprogrammed to differentparts, lines etc. and forms the basis of a generalized robot controlsystem.

What is claimed is:
 1. A method of assembling doors or other panels intoopenings in objects comprising the steps of:providing a robot withelectro-optical sensors on the tooling used to grasp said panel, withsaid sensors, sensing the location of said object and said opening insaid object, positioning said panel in said opening with said robotusing said sensors to feed back data to the control system of saidrobot, from said data, adjusting said position in said opening toequalize the gap between said panel and said opening so as to align saidpanel with said opening.
 2. A method according to claim 1 wherein saidsensors are further used to guide the robot to position said tooling incorrect relationship to said panel before grasping said panel.
 3. Amethod according to claim 1 further incorporating the step ofelectro-optically sensing targets on said object containing said openingto rough control the position of the robot to said object.
 4. A methodaccording to claim 1 including the further step of constraining at leastone axis of said object motion to limit displacement or velocity.
 5. Amethod according to claim 1 wherein said electro-optical sensor is aphotodetector array camera.